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for Advanced Learning in Integrated Systems Technology. 



STUDENT-INITIATED REPORTS: 
OPERATIONAL ANALYSIS IN THE EVALUATION OF CAI CURRICULA 



Michael J. HUleliohn 



Rationalf 

Opmtiontl analysis, a subset of formative evaluation, takes place while a course is 
in progrMs. Student-perceived problems, as reflected by student-initiated reporU, are the 
source of data for this phase of formative evaluation. As we use the teim here, evaluation 
provides daU to decision makers (Scriven, 1967). The difference between formative and 
summative evaluation bes in the nature of the decision and who makes it. Formative 
evaluation output is used mainly by course developers and administrators, whUe summa- 
tive data are the basis for potential consumer decisions (Scriven, 1967; Stake, 1960; 
Cunningham, 1971; Mehlinger and Patarick, 1970; Johnson, 1970). Formative evaluation, 
ttien, is a continuous cybernetic process tiiat begins after tiiere is an assurance that a 
student can get from tfie beginning of the instructional program to the end, and it may 
or may not end when the summative evaluation is initiated. 

Using the student's perceptions and actions to describe tiie instructional prosam as 
It actuaUy occurred, which may or may not be congruent witii the developer's prediction 
is a technique that has been proposed before (Fitxpatrick, 1970; Tracey, 196S; Yeacer 
and UndvaU, 1968; Mehlinger and Patrick, 1970). Some of tiie techniques that have been 
tried include having tiie evaluator (autiior, developer, etc.) be witii tiie student to obaerve 
him as he goes tiirough tiie instruction, employing tiiird-party observations, and 
debriefing. The techniques tiiat have evolved, however, have not satisfied aU the criteria 
of timeJmess, cost-effectiveness, accuracy, and relevance to tiie decision maker's needs. 
The techniques and tooU for operational analysis herein described have been successfully 
employed by Project IMPACT in a tutorial Computer-Administered Instruction (CAI) 
eivironment and meet these criteria. 

The placement of formative evaluation between the debugging (verification) of the 
instructional logic and the summative evaluation is shown in Figure 1. The derign and 
devetopment of the management (and evaluation) tooU take place after the characteristics 
of tiie instruction are specified, but before formative evaluation begins. The update 
controUer that receives it§ input from the formative evaluation effort is defined as a 
revision algorithm whose function is to indicate, to the course developers, where in tiie 
design and/or development of tiie instruction he has to return to make a change. The 
opentionaUzed course extends from module (topic) 1 to module N. Once students are 
taking the course, the formative evaluation process officially begins. 

The shaded blocks in Figure 1 represent our version of the usual analyses for 
formative evaluation. The last decision diamond on tiie right is superfluous if tiie 
module N and tiie course criterion tula are tiie same. The end-of-module analysis 
includes item-by-item and summary data on the student/instruction interaction. Test 
scores, latencies, question solutions, commento, expectancy operators, error rates, and 
anything else worthy of measurement are examined during this analysis. The data are 
massaged and, if deficiencies surface in tiie instiruction, are used to effect systematic 
improvement in the overall instructional program. When the module functions as pre- 
scribed, tiie data are stored so tiiat if a later module, for which module 1 is a 
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prerequisite, malfunctions, these daU wUl be re-examined. Also, data gathered in 
eJlik!ltton*^"***°" ™«y be useful Uter to a decision maker performing a summative 

However, if the role of evaluation is to provide data for decision making leading to 

>"JP«>vement of the instructional environment (Scriven, 1967; Lindvall and 
Cox, 1969; Jones, 1971), then tne end-of-module a;ialysis does not provide enough 
mformation m individualized instruction. To be complete, evaluation must include more 
than the usual quantitative measurement (Jones, 1971; Yeager and UndvaU,1968) 
impUed by the performance data withered at the end of modules. The qualitative 
nonnumenc measurement via student-initUted reporta made during dUplay-byHlisplay 
mteraction u a potential source of valuable formative information. 

One reason for inhibition on the part of developers to making curricular changes is 
thai, as Larkins and Shaver (1968) found, standard evaluation deugns oftan do not 
provide adequate information to justify and then determine curricular modifications This 
» especially appUcable in nontraditional types of instruction. Minor modifications are 
often required to meet the needs of subpopulations within a student group and 
asseument of these needs is often best made by members of the subpopulation. 

The goal of the instructional situation determines the relevant measures for evalua- 
tion of the mstruction (Flanagan and Jung, 1970). If the goal is individualized instruc 
tion, then the mdividual is an important source of data and should not be ignored. 4n 
order for a decision maker to reach conclusions that are appropriate in this situation, he 
should have feedback firom the students who undergo the instruction. Therefore another 
type of analysis should take place during formative evaluation and, since it is concurrent 
m time with the operationalized course, it is caUed operational analysis. 

During operational analysis, student-perceived problems are entered into the data 
base as they occur and decisions are made whether to take action immediately or to use 
the data m the end-of-module analysis. As stated by Seklel (1971, p. 5), the premise 
behmd this approach is that "The organism [student] is an active organizing force in iU 
mtwaction with iU environment. . Through our self-reporting or student-initiated 
techniques, we are able to capitaUze on these active, organizing capabUities. Individual 
differences m student/environment interactions can be revealed and the individual 
student's perceptions ran be used to improve the instructional program. 

Staffing and Physical Characteristics of Learning Area 

Before going into the techniques themselves, it is desirable to describe the personnel 
and peripheral hardware configuration in the Ptoject IMPACT learning environment Key 
personne during course operation are shown in Figure 2, with course administration 
personnel m the section enclosed by dotted lines. 

A brief description of the functions of each member of the latter group is in order 
The Director for Daily Operations is responsible for the efficient day-to-day "running" of 
the course. He serves as liaison to the directors for other components of the total system 
It 18 his responsibUity to ensure that required system (any component) jchanges are 
accomplished m such a manner that student progress is minimally affected. In effect, he 
18 a traffic manager for problem solutions and computer utiliz^ion while studenU are 
on-lme. His duties also include supervision of the proctoring staff. The Operations 
Monitor(OM) is responsible for the smooth running of a particular student period/shift 
(Project IMPACT had three daily three-hour shifta) and supervising the proctors on t'.iat 

^ ^ ^nt^P^ted from Hein* von Foertter. "Molecular Ethology: An Immodest Proposal for Semantic 
Clarification, Molecular Mechanism$ in Memory and Learning, O. Ungar (ed.). Plenum Press. New York 
1J*70. p. 234. * 



ERIC 



3 



Courtt AdminHtration PanofiMl 

r » 



Oirtctor for 

IMIy 
Oparctiom 



I 



Optratiom 
Monitor 
(Shift Supirvitor) 



Proctor I 



Director for 
ImtnietioMl 
Dttign 




Diroetor for 

InttmotioMl 

^ — — 

UOmvfn 



Diroetor for 

•OflWWt 



FiJ^rt 2. K«y ParionntI During Courw Optratkm 



tnift. He lervet as a resource for the proctors and is the only person asidgned to the shift 
who can give the student subject-matter relevant information. Because of these functions, 
it is desirable that the OM take the coune he is "OMing" for, and also serve some time 
as a proctor. These experiences give the OM first-hand knowledge of the kinds of 
problems encountered on "both sides of the fence." 

The proctor's function is purely procedural and the students are informed of this. 
He is responsible for ensuring that the student has all required materials, correcting minor 
hardware difficulties (such as auxiliary visual display does not align properly with primary 
display), answering system-genemted proctor calls, diagnosing the student's problem and, 
if it is subject-matter specific, referring the problem to the OM. The continuous inter- 
action of the student with the instruction is shown in Figure 3. It is included to show 
that after the proctor/student interaction occurs the student returns to his interaction 
with the instruction. It is not the proctor's function to intercede for the student, but 
rather to enable the student to continue on his own. 

Because of the large amount of proctor/student interaction that may occur, and 
because the students perceive the proctor as a course and system expert, a prospecthre 
proctor is required to go throu^ the course of instruction in the same manner as any 
regular student. It should be noted that no mention was made of a requirement that th« 
proctors (or OMs) be subject-matter experts. They must be course experts (mechanics and 
texts), but at Project IMPACT the courseware subsystem has enou^ off-lbie documenta- 
tion (see Willis, et. a/., 1972; Hillelsohn, 1974; and various support documents, internal 
proctor guides, memos, etc.) that subject-matter expertise is not requisite for proctors. 

Proctors must be knowledgeable about the peripheral hardware used in the learning 
environment. The layout of the Project IMPACT learning environment is unique in that 
the student carrels are designed so that the student is, in effect, in his own private room, 
with all the necessary learning materials at his disposal. The learning area is diagrammed 
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Proctor 
inttrvention 



Figun 3. Proctor's Position in the 

Studtnt/lnttniction Interaction 

in Figure 4, witli the m^r hardware componenU identified. ProctoA must be familiar 
enou^ with th*» cathode ray tube (CRT) to solve such problems as locked keyboards 
scrambled screeiw, flicker, and so forth, and to recognize the symptoms of problems that 
require a fkcton; service mechanic V fix. With the auxiliary vUual device, problems that 
should be remedied by the proctor include realignment of visual images on the display 
screen, reloadmg of film, and ensuring proper coordination of primary and auxUiary 
display. The student initiates proctor intervention for the solution of these and other 
problems by pressing the proctor call button, at which Ume a buzzer sounds in the 
pro^r room and a li^t, d^ignating the carrel from which the request originated, is 
Uluminated on the call box. The proctor then goes to the student, diagnoses the student's 
problem, and ejects a solution. 

Data on student reaction to proctors and how their functions were performed were 
gathered m the questionnaire that aU itudenU fiUed out upon course completion. The 
resulU show that 76% of the studenu perceived the proctors as friendly, 80% perceived 
?Ml knowledgeable, 77% as helpful, 97% as maintaining a proper degree of formality 
and 77% as able to supply information as required. In short, non^ubject matter experts 
can be successfuUy employed to interact with studenU during instruction. 

Th« Rtport 

In a tutorial CAI application it is crucial that the course administrator, course 
evaluator, and others be able to determine what occuned during the proctor-student 
mtemction: Has the student come across a problem that wUl affect other studenu when 
they reach that point in the course? Has the student asked the proctor for information 
that has already been presented or is going to be presented? Has the proctor given the 
student too much subject matter relevant support? Or has the proctor given him 
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misleading information? The possibilities are endless, and if each interaction is not 
documented, the end-of-module evaluation of both course and student performance is 
confounded by an unknown variable. 

Therefore, at Project IMPACTT, a Trouble Report (TR) was designed and developed 
in which the proctor can document his interaction with the student. After an off-line 
evaluation period, the TR was put on-line in iU present form (shown in Figure 5). The 
proctor uses the CRT in the proctor room to input the daU, and is "signed-on" to an 
administrative course (VOYEUR, Appendix A) as a student, thereby not increasing 
system overhead. The records generated are compatible with aU other student records, 
but are identified as proctor recordings by the unique course name. The display was 
designed by both an instructional and an appUcations programmer so that the data would 
be recorded in such a form as to allow easy manipulation and analysis. 

The data are first recorded in the same format and manner as any student record 
(see Figure 6), but bytes 205 through 304, instead of containing the student response, 
contain the information that the proctor u reporting.' What is this information? When 
the proctor leaves the student and reenters the proctor room he goes to the CRT and 
calls up the TR element. He keys in the CWIP, or location in the course where the 
interaction occurred, then the student number, cubicle (carrel) number, his initials, the 
initials of the OM for that shift, and then the alphanumeric code defining the trouble 
that prompted the student caU. The student group identification number, date, time, and 
TR number (ID) are filled in by the software system. 

There are four major categories of troubles shown in Figure 6. Those listed under 
the heading "IP" relate to student-perceived problems with the instructional text; those 
under "CW" relate to problems in the instructional logic and related software; 
"HRDWRE" problem entries recorded hardware failures that are more often related to 
the cubicle wherein they occurred than to the student who initiated the call; problems 
listed under "MISC." are mainly for record keeping on problems which are usually fixed, 
if fixing is required, by the proctor during the proctor call. Trouble "50" allows the 
proctor to define the reason for the proctor call if it does not fit into one of the 
predefined categories. The specific problems listed on the Trouble Report are, in many 
cases, unique to the Project IMPACT hardware/software subsystem (documented in 
Underbill and Stelzer, 1972; Gameau, etai, 1972; Shuford and Stelzer, 1972) and would 
be replaced in other appUcations with relevant entries. The second page of the TR 
(shown in Figure 7) allows the proctor to explain and describe the interactfon he just had 
with the student and, if relevant, provide suggestions for action to eliminate the reason 
for the proctor call. The information on the top of the page is duplicated, by the system, 
ftrom page 1 (Figure 5) of the TR onto its appropriate position on page 2 (Figure 7) and 
the proctor need fill in only as many lines of explanation as are required to fully explain 
what occurred during, and what caused, the interaction. 

As stated earlier, these recordings are in the same format as any student record, but 
bytes 205 through 304 contain the encoded information that the proctor entered into 
the system. The record layout for this segment of the recording for page 1 of the TR is 
shown in Figure 8. For recordings of page 2, these bytes were filled with exactly what 
the proctor says in his explanation. Byte 304, the update code, is not entered by the 
proctor, but is most important to the course administrator. 



For detailed explanation of the record entries see Willis and Stelzer, 1972, 
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Administration 



Once a day the data in the TR file are reproduced in hardcopy and referred to the 
directors of the various affected activities (see Figure 2) for action* The nature of the 
problem determines which director is designated as the referee for that TR« A referee has 
three possible courses of action: 

(1) If he feels that the problem/interaction is unique and not likely to recur, 
he can choose to take **no action.** 

(2) If all (or most) of the students traversing the point in the course where the 
problem/interaction occurred are likely to be affected, he would rectify the 
problem immediately. 

(3) If the referee is unsure as to how many students would have difficulty with 
the reported problem he can table any action, pending receipt of addi- 
lional TRs. 

After the referee has taken action (1 or 2 above) on a TR, he writes either **fixed** 
or ^*no action required** on the TR and returns it to the proctor room. The proctor then 
keypunches a card for the TR, with the TR identifier and sets a status flag by punching, 
in column 11 of the card, l; which means the TR is invalid; 2, which means no action is 
necessary; or 3, which indicates that the' problem has been fixed. Cards are not punched 
for TRs that are not returned and a default value of 0 is set as the update code. The data 
on the punched cards are then merged with the TR file data and the status flag is placed 
in byte 304 of the TR record as the update code. 

At the end of a meaningful period of time (at Project IMPACT, after each student 
group has completed the course), the course administrator calls for a readout of all TRs 
that have an update code of 9-in other words, those TRs where action was tabled. He 
then checks with the referees to ensure that some action is taken on the reported 
problem. With this system, course administrators ensure that the individualized instruc- 
lion is responsive to the expressed needs of the individual student. 

However, students also have needs that they may not recognize. For example, if a 
question is not scored correctly the student can be penalized without realizing that this is 
happening. One of the duties of the proctor is to examine the formatted student record 
for scoring and recording errors during the end-of-module analysis. If such an error is 
found, the proctor again goes to the TR and enters an S8 (or what we call IP internal 
report). This is a non-student-initiated report, but it goes through all the procedural steps 
of a regular TR, with the proviso that the referee is specified by the course administrator. 
This type of TR must result in an update code of 3 within a minimum amount of Jtime. 

Applications 

Any one of the proctor entries can be treated as a variable by the course adminis- 
trator. For example, if he suspects that a specific auxiliary visual device is giving an 
inordinate number of problems, he can request an output that would show a frequency 
count of all auxiliary visual related TRs, sorted by cubicle. The output would either 
confirm or deny his subpicions and appropriate action could be taken. Therefore, the 
administrator can get output based on any entry or combination of entries on the TR, 
such as specific troubles, locations in the course, student, proctor, cubicle, time of day, 
and BO forth. Because the TR is dynamic (can be added to at any time), a listing of all 
trouble number 508 is usually beneftcial to the administrator at specified points in time. 
If there are numerous occurrences of a specific problem in this listing, it is desirable to 
add it to the regular list of troubles, and update all previous occurrences with its new 
alphanumeric code. Thereby, it can be manipulated in the same manner as any 
ottter trouble. 
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Another type of end«of-group report is the Trouble Report Summary, in which the 
alphanumeric code next to each trouble is replaced by a frequency count for that 
trouble. In the example shown in Figure 9 there is, let us say, an unacceptably high 
instance of system problems for the student group being reported on in the summary. 
The course administrator would inform the directors of hardware and software of the 
unacceptability of the situation and they are expected to rectify it. The efficacy of the 
solution implemented can be checked by continuing to get frequency reports during and 
after the next student group* 

In fact, summary TRs are very helpful for assessing the success of a previous analysis 
and subsequent revisions. Summary data over five student groups for selected instruc- 
tional and logic programming related troubles are shown in Figure 10. Between Groups 
11 and 12, the course reported on underwent substantial revision. Two data points are 
reported in this summary (frequency and mean number of TRs per student). Because the 
groups have unequal Ns, the second figure, mean number of TRs generated per student, is 
more meaningful to examine than the totals. 

Group 
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1.00 


1.43 


0.17 


0.48 


Code « GS 












Frequency 


1.00 


1.00 


1.00 


0.00 


0.00 


Mean/i 


0.17 


007 


0.07 


0.00 


0.00 


Code « G7 












Frequi^ncy 


5.00 


1.00 


2.00 


0.00 


1.00 


Mean/i 


0.83 


Oi07 


0.14 


0.00 


0.06 


Frequency 












Total (Q) 


19 


17 


23 


4 


11 



Where: A4 « CRT Text No Good G1 = Response Analysis Problem 
A7 - Direction Unclear G5 - Wrong Branch 

AS - Subject Matter Question G7 « Wrong Feedback 



Figure 10. Summary of Tiuuble Reports-Groups 11 thru 15 
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This particular example would be interpreted by a course manager (or evaluator) as 
reporting that the revision to the instructional text is successful, but that the revisions 
caused problems in the instructional logic that were not solved untU after Group 13. With 
only smaU exceptions, the mean number of TRs per student was reduced bom group to 
ffroup. The most striking reduction in TRs is in category A7, "directions unclear." This is 
the type of problem that would not normally surface during the end-of-module analysis, 
but 18 empiricaUy evident during the operational analysis when student problems are 
documented via the TR. It is also evident from Figure 10 that as the course is "polished" 
fewer student-initiated interactions occur and therefore reduced proctor staffine 
18 possible. " 

Staffing requirements as a function of TR daU can also be determined by the 
characteristics of the student population. Figure 11 shows data on the number of 
student-initiated reporte over two divisions of a course (Hillelsohn, 1974) for high and 
low performers (equal Ns for aU ceUs) in each of the divisions. It U important to note 
that the high and low performers were chosen, on the same criteria, separately for each 
division. Although there is some overlap, each cell is made up of a unique group of 
students. Only the criteria for assignment were constant. All system-initiated reports were 
eliminated for this summary. The data show that low performers initiate more Trouble 
Reports by about two to one than high performers for this course. To the course 
administrator this means that if there is a group of predicted low performers about to 
take the course, more proctors would be needed than if there is a group of predicted 
high performers. An alternative course of action would be to take this information to 
course evaluators/developers and have them revise the course so that the incidence of low 
performance is reduced, or raise the entry level requirements for the students. 



Studtnt Groups (N"20 ptr group) 



oe 



\^ High A 


Low A 


High B 


Low B 


Div A 


17 


64 


21 


59 


Div B 


31 


44 


25 


43 


00 

i 


48 


108 


46 


102 



Figura 11. Number of Reports Initiated 
by High and Low Performers 



(inclusion 

Student-initiated reports proved themselves to be an important data source for 
decision makers, both for course development and for administration. They are timely, 
because they enable the decision makers to get daily reports and effect prompt repairs if 
necessary. The reports are entered into the data base when the CAI hardware/software 
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system is already operating for students and there is no attendant cost to inputting the 
operational analysis data. The only additional cost to the CAI system overhead is the cost 
of reporting the data, which is done via batch programs. Since it is the individual student 
who is initiating the report, the data are more likely to accurately reflect student 
perceptions of problem areas in the instruction and thereby supply an additional relevant 
dava point for the decision maker who has to evaluate an individualized learning system. 
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Appendix A 



VOYEUR 



COURSE NAME: VOYEUR 

VOYEUR is a course only in the sense that the text is in the instructional text 
nies and the logic is m the instructional logic files. It is really a support program to 
aid proctors and course managers in performing their duties. 

In Project IMPACT, the VOYEUR program was used on a proctor terminal to 
monitor the progress of a group of on-line studenU. The four main elemento of the 
program are described below: 

(1) gtudent Status. Lists each active student by number, with the following addi- 
tional information: (a) location in the course (by course label), (b) most recent quiz 
score (as CTR 29 - ), and (c) last input to the system (as Buffer 0 « ). ThU information 
u retrieved from each student's line daU area and updated every time the proctor 
executes a PRESS SEND on the monitor terminal. 

Figure A-1 shows a porUon of VOYEUR output on student status. Retrieving 
inTormation from each student's line data area, the program can report on 10 studente 
at a time. 



03/26/73 16:26 










STUDENT - 


255 


LABEL ' 


BIA16A 


CTR 29 = 0 


BUFFERO»SIGN OFF 








STUDENT = 


282 


LABEL « 


BHAOOA 


CTR 29 = 177 


BUFFERO=LINK B 









Figun A h Portion of VOYEUR Output 



(2) Trouble Report. Presents a form to be completed by the proctor after each 
interaction with a student. As shown in Figure A-2, the information to be supplied by 
the proctor includes the following: ^ 

CWIP-course location where interaction occurred. 

STUDENT NO.-identification of student needing help. 

CUBICLE-student station where interaction occurred. 

PROC— initials of the proctor answering the call. 

OMHnitials of the chief proctor of the shift. 

o5Sp?^^~*?P**'"""*"*' °' P°"'*»'* problems presented in four categories 
OTHER-explanaUon of the trouble code "50". 
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AddiUonal blank lines are also provided for more explanation if desired. The identifi- 
aition number (ID.) of the trouble report is supplied by the system, along with the date, 
Hie time, and the code of the student group (this is TR12 in Figure A-2). 

In Pro^ct IMPACT, copies of these trouble reports were referred to appropriate 
*!fi i?^ u °' eliminating the problems. ThU Referee Action was also 

added to the trouble report on-line. AU daU entered by proctors, referees, and the system 
were incorporated mto a data base for output by variables such as number of trouble 
reporU per student, per course label, per type of problem, etc. 

(3) System Status. Pre-formatted command of the executive program (Zeus) that 
aUows a proctor to inspect the status of the system, including the latest messages on the 
console, the size of the partitions, and the retative activity of each partition in the 
mam computer. 

(4) Display. Pre-formatted command of the executive program (ZEUS) that allows 
a proctor to quickly look at any element in the course. 

The pro'itor can move easily from one element to another in VOYEUR by entering 
on the CRT screen a one<:haracter designator that initiates an immediate branch to 
the element desired. These designators are: 

V for Student Status, 
T for Trouble Report, 
S for System Status 

Y for Display. 
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